ISIA at the ImageCLEF 2017 Image Caption Task

This paper describes the details of our methods for participation in the caption prediction task of ImageCLEF 2017. The dataset we use is all provided by the organizers and doesn’t include any external resources. The key components of our framework include a deep model part, an SVM part and a caption retrieval part. In deep model part, we use an end to end architecture with Convolutional neural network (CNN) and a Long Short-Term Memory (LSTM) to encode and decode images and captions. According to the statistics of training dataset, we train different models with different lengths of captions. Then in SVM part, we use Support Vector Machine (SVM) to determine which model to use when generating the description for a test image. In this way, we can combine these models from the previous deep model part. In caption retrieval part, we use the image feature extracted from CNN and apply Nearest Neighbor method to retrieve the most similar image with caption in the training dataset. The final description is the aggregation of the generated sentence and the caption retrieved from the training dataset. The best performance of our 10 submitted runs ranks the 3rd in group which doesn’t use external resources.